<html>
<head><meta charset="utf-8"><title>aarch64 self-hosted won&#x27;t start · t-infra · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/index.html">t-infra</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html">aarch64 self-hosted won&#x27;t start</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="216153573"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216153573" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216153573">(Nov 09 2020 at 22:56)</a>:</h4>
<p>The aarch64-gnu job hasn't started for two PRs in a row.   Is the server maybe down?</p>



<a name="216164040"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216164040" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216164040">(Nov 10 2020 at 01:32)</a>:</h4>
<p>cc <span class="user-mention" data-user-id="120823">@DPC</span> FYI, the rollup is going to fail.  If the next PR doesn't kick off, I can close the tree until someone can investigate.</p>



<a name="216164049"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216164049" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> DPC <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216164049">(Nov 10 2020 at 01:32)</a>:</h4>
<p>the same issue as the previous pr?</p>



<a name="216164059"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216164059" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216164059">(Nov 10 2020 at 01:33)</a>:</h4>
<p>Yea, for some reason the aarch64 runner isn't starting.</p>



<a name="216164136"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216164136" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> DPC <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216164136">(Nov 10 2020 at 01:34)</a>:</h4>
<p>closed the tree</p>



<a name="216192028"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216192028" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216192028">(Nov 10 2020 at 10:14)</a>:</h4>
<p>ugh</p>



<a name="216192590"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216192590" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216192590">(Nov 10 2020 at 10:21)</a>:</h4>
<p>omfg</p>



<a name="216192828"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216192828" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216192828">(Nov 10 2020 at 10:24)</a>:</h4>
<p>ok I forgot to push the commit that bumped the github actions runner</p>



<a name="216192876"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216192876" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216192876">(Nov 10 2020 at 10:24)</a>:</h4>
<p>so the runner deadlocked trying to update (I forcibly disabled updates otherwise our fork would get overridden)</p>



<a name="216210557"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216210557" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216210557">(Nov 10 2020 at 13:31)</a>:</h4>
<p>ok sent a mail to our github contacts about this</p>



<a name="216210649"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216210649" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216210649">(Nov 10 2020 at 13:32)</a>:</h4>
<p>it's not fair to expect us to rebase the fork of the runner and to rebuild the images in 4 hours since the new runner tag was pushed to github</p>



<a name="216217808"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216217808" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216217808">(Nov 10 2020 at 14:22)</a>:</h4>
<p><span class="user-mention" data-user-id="121055">@Pietro Albini</span> in the meantime, do we have docs on how to do so? I think just rebasing the repository should work, right?</p>



<a name="216217866"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216217866" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216217866">(Nov 10 2020 at 14:23)</a>:</h4>
<p>(since we check every 15 minutes for a new version)</p>



<a name="216218067"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216218067" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216218067">(Nov 10 2020 at 14:24)</a>:</h4>
<p>the two steps needed are:</p>
<ol>
<li>rebase <code>rust-lang/gha-runner</code> on the latest tag, changing the version number adding a <code>-rust1</code> at the end of it</li>
<li>once the CI for <code>rust-lang/gha-runner</code> finishes, change the version number in <code>images/ubuntu/scripts/install-gha-runner.sh</code> in <code>rust-lang/gha-self-hosted</code></li>
</ol>



<a name="216219685"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216219685" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216219685">(Nov 10 2020 at 14:35)</a>:</h4>
<p>Ok!</p>



<a name="216941463"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216941463" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Gaynor <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216941463">(Nov 16 2020 at 22:25)</a>:</h4>
<p>Looks like this is maybe happening again <a href="https://github.com/rust-lang-ci/rust/runs/1408579929">https://github.com/rust-lang-ci/rust/runs/1408579929</a> ?</p>



<a name="216941643"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216941643" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216941643">(Nov 16 2020 at 22:27)</a>:</h4>
<p>cc <span class="user-mention" data-user-id="121055">@Pietro Albini</span></p>



<a name="216941676"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216941676" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216941676">(Nov 16 2020 at 22:27)</a>:</h4>
<p>ffs</p>



<a name="216941880"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216941880" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Gaynor <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216941880">(Nov 16 2020 at 22:29)</a>:</h4>
<p>Sorry I noticed :-(</p>



<a name="216941979"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216941979" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216941979">(Nov 16 2020 at 22:30)</a>:</h4>
<p>ok it was still blocked because of the forced self-update</p>



<a name="216942039"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216942039" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216942039">(Nov 16 2020 at 22:31)</a>:</h4>
<p>should be fixed now</p>



<a name="216942096"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216942096" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Gaynor <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216942096">(Nov 16 2020 at 22:31)</a>:</h4>
<p>How long does that builder take? Is it likely to succeed in the allotted time window for bors, or would be it be better to just restart?</p>



<a name="216942119"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216942119" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216942119">(Nov 16 2020 at 22:32)</a>:</h4>
<p>2hr</p>



<a name="216942200"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216942200" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216942200">(Nov 16 2020 at 22:32)</a>:</h4>
<p>retried, good call <span class="user-mention silent" data-user-id="130046">Alex Gaynor</span></p>



<a name="216942270"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216942270" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Gaynor <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216942270">(Nov 16 2020 at 22:33)</a>:</h4>
<p>Glad to see my time spent tending to work's CI, various OSS project's CI, etc. have some transferability.</p>



<a name="216942301"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216942301" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216942301">(Nov 16 2020 at 22:33)</a>:</h4>
<p>this time the image was correctly rebuilt, but the <a href="https://github.com/rust-lang/simpleinfra/blob/bb1db33bad78b832de4fbd496e903fc9db522145/ansible/roles/gha-self-hosted/templates/build-image.sh">script that rebuilds the images</a> didn't restart the VMs</p>



<a name="216942395"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216942395" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216942395">(Nov 16 2020 at 22:34)</a>:</h4>
<p>so those VMs were stuck always running and always rejecting runner upgrades</p>



<a name="216942436"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216942436" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216942436">(Nov 16 2020 at 22:34)</a>:</h4>
<p>tomorrow I'll have to change the script to restart idle VMs when an image update happens</p>



<a name="216942926"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216942926" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216942926">(Nov 16 2020 at 22:39)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> fyi, what I did to investigate the issue, so I'm not a point of failure for aarch64 CI:</p>
<ol>
<li>connected to <code>ci-arm-1.infra.rust-lang.org</code> through the bastion</li>
<li>checked the status of the VMs, they were running (<code>sudo systemctl status gha-vm-{host}-{num}</code>, for example <code>sudo systemctl status gha-vm-arm-1-1</code>)</li>
<li>since they were running, logged on a VM (<code>ssh -p 220{num} manage@localhost</code>, password <code>password</code>)</li>
<li>checked the status of the gha runner inside the VM (<code>sudo systemctl status gha-runner</code>), log said "blocked self-update"</li>
<li>"ffs"</li>
<li>logged off the VM, checked that the image was rebuilt recently (<code>sudo systemctl status gha-self-hosted-rebuild-image</code>)</li>
<li>restarted all VMs on the machine (<code>sudo systemctl restart gha-vm-arm-1-1 gha-vm-arm-1-2</code>)</li>
</ol>



<a name="216943295"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216943295" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216943295">(Nov 16 2020 at 22:42)</a>:</h4>
<p>thanks!</p>



<a name="216943307"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216943307" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216943307">(Nov 16 2020 at 22:42)</a>:</h4>
<p>I just am busy tonight so didn't try to investigate</p>



<a name="216943377"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/216943377" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#216943377">(Nov 16 2020 at 22:43)</a>:</h4>
<p>no worries</p>



<a name="217480691"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aarch64%20self-hosted%20won%27t%20start/near/217480691" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Nelson J Morais <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aarch64.20self-hosted.20won&#x27;t.20start.html#217480691">(Nov 21 2020 at 02:37)</a>:</h4>
<p>just a heads up. I've started working on a collector to track the status of the gha runners that eventually will help identify this kind of issues with staled runners earlier .<br>
WIP PR <a href="https://github.com/rust-lang/monitorbot/pull/13">here</a></p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>